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Abstract — Low-density parity-check (LDPC) codes on symmet- 
ric memoryless channels have been analyzed using statistical physics 
by several authors. In this paper, statistical mechanical analysis of 
LDPC codes is performed for asymmetric memoryless channels 
and general Markov channels. It is shown that the saddle point 
equations of the replica symmetric solution for a Markov channel 
is equivalent to the density evolution of the belief propagation on 
the factor graph representing LDPC codes on the Markov channel. 
The derivation uses the method of types for Markov chain. 

Keywords — Low-density parity-check codes, Markov channel, 
replica method, method of types, large deviations. 

1 Introduction 

The repUca method is a tool for the evaluation of free en- 
ergy in statistical physics. Although the mathematical rig- 
orousness of the replica method has not been proved, the 
replica method has been used not only for problems of sta- 
tistical physics but for problems in several areas including 
information theory Q. The analysis of low-density parity- 
check (LDPC) codes using the replica method is shown for 
memoryless symmetric channels by Murayama et al. [2J and 
Montanari Q. After some period, LDPC codes on binary- 
output asymmetric channel is analyzed by using the replica 
method by Neri et al. L4 1 . On the other hand, LDPC codes 
on channel with memory is also of great interest in wireless 
communication and magnetic recording |5 |. LDPC codes 
on restricted types of finite-state Markov channel are also 
analyzed by Neri and Skantzos ||6| . In this paper, we analyze 
LDPC codes on general asymmetric memoryless channel 
and general Markov channel by using the replica method. In 
the analysis, we use the method of types for Markov chain 
in order to generalize the method recently obtained in I?!. 

Despite lack of mathematical rigour, the replica method 
is known to yield rigorous bounds in some cases ||8l, ll9l. 
Moreover, a tight bound of MAP threshold, which is derived 
in an elegant way in |10|, is found to empirically coincide 
with the result of the replica method. 

2 Memoryless channel 

2.1 LDPC codes and the number of codewords 

In this paper, we deal with regular LDPC codes. Gen- 
eralization for irregular LDPC codes is straightforward as 
in [7J . We consider factor graph (Tanner graph) including 
variable nodes and A^(l —R) factor nodes (check nodes). 
The degrees of variable nodes and factor nodes are / and r, 
respectively, for which the relation R= X — ljr holds. Edge 
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connection is randomly chosen uniformly from {Nl) \ pos- 
sible connections. Let A" = {0, 1} and f{x) : X'' — > {0, 1} 
be a factor function taking 1 when the number of 1 s in a: is 
even, and taking otherwise. Let x^^ be the neighborhood 
of a factor node a. The uniform distribution on codewords 
X € of the LDPC code defined by the Tanner graph is 
given by 

Here, Zq is a constant for the normalization, which in this 
case is the number of codewords in an LDPC code. Let 
E[ ] be the expectation with respect to the edge connections. 
In Q, the following lemma is shown for general f(x). 
Lemma 1. 

lim -logEfZnl 

= max jlogZv + -logZf-/logZe| (1) 

where TZ denotes the set of saddle points of the function for 
which the maximum is taken, and where 

r 

Zv := 52 '"f-^vW', Zf:= ^ /(jr)]^OTv^f(ji:,) 

xex xex> i=i 

Ze := ^ mf^^{x)m,j^f(x). 
xex 

The saddle point equations determining the two probability 
functions mv^f(x), m{^y{x), both defined on X, are 

OTv^f(x) oc mf^^(x)''^ 

mi^^{x)oc^Y^ f{x) Y\ my^f{xj). 

i=lxeX'\xi = x j=\,j=/=i 

In the case of regular LDPC codes, it is proved |TD 
that the maximizer for the right-hand side of ([T]i is given 
by mv^f(jc) = mi^^{x) = 1 /2, for which 



lim -logE[Zo]=/?. 



(3) 



Since each independent factor reduces the number of code- 
words by half, the inequality (1 /N) logZo > R always holds. 
Hence, for regular LDPC codes. 



lim -EflogZol =7? 



and for any e > 0, 
1 



log Zo < 7? + e , with probability 1 - o ( 1 ) . (4) 



The result of this paper can be generaUzed to any irregular 
LDPC code ensemble satisfying ([3j. 



2.2 Replica analysis 

In this subsection, we consider LDPC codes transmitted 
on an asymmetric memoryless channel. Let y be the output 
alphabet and ^^(y | jc) be a probability of an output y €i y 
when X ^ X is transmitted. The a posteriori distribution of 
a codeword x G given an output y E y^ is 

/'(^l3'):=^n/(^5«)n^(3''-|-^0 

^ a ;=1 

I n/(^5«)n^(3'.-u.)- 

JCGA'" « '=1 

Let E[-] be the expectation with respect to the output 3; G y'^ 
of the channel, which obeys 



1 



The aim of this paper is to evaluate 

lim(l/A^)E[E[logZ]] 



(5) 



which gives the expected conditional entropy of codeword 
given an output |12|. The quantity (|5]l is evaluated via the 
replica method. First, from (|4|i, Zq in the definition of p{y) 
can be replaced by 2^^*. Then, one obtains 



E (Yim-D nnnyi\-\ 
e[z"]=epw e (nfif(4l)]hhny.\-r) 

yey'^ xe{x")'^ \ " k=i ) i=ik=i 

E nhn-fl) 

i=l \ysyk=0 ) 

where x^^^ g X and z'*^' G Af' denote the variables corre- 
sponding to ^h replica of G and z G (<%'"+')'', re- 
spectively. By using the method in Q, for n G {0, 1,2,...}, 
one obtains 

lim -logE[E[Z"ll 

= max I logZv + -logZf — /logZe 1 — 

(mf^,(jr),m,^f(jr))GK ^ J 

(6) 

where denotes the set of saddle points of the function for 
which the maximum is taken, and where 



The rephca symmetry (RS) assumption that we use is 

my^i{x) = mv^f(xo) / ]~[Mv^f(x;)d<I>(Mv^f | jcq) 
( =1 

/■ " 

m^^x) = mf_^v(-«o) / n^f-^v(jc;)d4>(Mf_^v | xq) 
( =1 



(7) 



where <I> and <i> are distributions of measures on X. In O, 
the notation .jc, is used instead ofx^'^ for / G {0, . . . ,n}. From (O, 

in the right-hand side of (|6|l can be replaced by the right- 
hand side of ([T]). On the RS assumption (|7]|, we obtain the 
following lemma. 
Lemma 2. For any n G M, 



lim -logE[E[Z"ll 

= ^ extr |logZv(n) + -logZf(n) — ZlogZe(«) 

where extrx{-F(x)} denotes F{x*) where x* is the saddle 
point of F{x), and where 

^v(«):=E ^"^;f Ly E^(^l^) 

nd4>(M«^ \x)(Y^W{y\ x){{M^;l{x)\ 

i=\ \xex 1=1 / 

Zf(n) - V .f{x)XYi=\m^^f{xi) 

Yld<P{Mi;l,\x,) ( f{x)flMi-l,{x^)] 

i=l \xex' 1=1 / 

Ze n := 2^ / \ / ^ / d1> Mv^f x 

■ d<I>(Mf_^, ' ( 1^ Mf_^^(x)M^^f(x)^ . 

From this lemma, the RS solution is obtained straight- 
forwardly Q. 
Theorem 3. 

^im iE[E[logZ]] . hm i lim 1 logE[E[Z«]] 
= extr {L £ /(^)nU^v^f(^,-) 



nd<I>(M('2,f|x,)log fix)YlMl%ix.)] 

i=l yj-GA"' !=1 / 



Zv:= E En^(3'UP) -f-vW 
xGA-'+i \yeyk=o ) 

^■■^ E ri/(^<'^)fi«w(^,-) 

x<^(X«+^ Y k=Qi 1=1 

Ze:= ^ mf_^v(-«^)»^v^f(-«^)- 

ji;GA'''+l 



•log E^(3'U)n^iw 

\.VGA i=l / 

' ' E ( \ I \ \ A^v^f 

Lxex'ni^y(x)m^^f(x) J 

•d4>(Mf_^v I ^)log I £ Mf_^v(^)Mv^f(x) j 



where the saddle point equations are (O and 



mf^^{x)^{Mf^^ \x) 
1 '' 



/(-«^)n;=l,//A:'Wv^f(-^0 



t\ xe:^x,=x ^xexr f{x) PI ■= i ,i^k my^fi^i) 



n d4>(Mi'2,f|x,) 



■5 Mf_^v, 



«l>(Mv^f \x)=Y,Wiy\x) Yld^M^'l^ I x) 



i=l 

A A') 



d Mv^f , ■ 



w(y|x)n;.i;M^:^,(x) 

L.rsxW{y\x)U'<ZlM^^,{x 



As mentioned above, all mf^y{x) and mv^f(x) in Theo- 
rem[3]can be replaced by 1/2. The saddle point equations in 
Theorem [3] are equivalent to the density evolution of belief 
propagation for asymmetric memory less channel |13|. Al- 
though the RS assumption is sufficient for symmetric chan- 
nels |3|, for asymmetric channels, 1-step replica symme- 
try breaking (IRSB) should be considered generally |4|. 
The equations for IRSB solution are obtained straightfor- 
wardly from ^ and the IRSB assumption. However, nu- 
merically solving the saddle point equations for IRSB solu- 
tion is much more elaborate. For LDPC codes, there exists 
a frozen solution which is the minimum of the RS solution 
with respect to temperature [3], [14], L4J. While the frozen 
solution can be relatively easily obtained Uke the RS so- 
lution, it is not obvious whether the frozen solution is the 
correct solution ||4J. 

3 Markov channel 

In this section, we consider LDPC codes on the gen- 
eral Markov channel. The space of states is denoted by 
S. The state-dependent channel is denoted by ^(y | x,s), 
and the transition probability of states by V{s' \ y,x,s), for 
y ^ y, X X, and s, s' G S. The probability distribution of 
the initial state s E S is denoted by Vo(*)- When the tran- 
sition probability is independent of y, we call the channel 
an intersymbol-interference channel. When the transition 
probability is independent of y and x, we call the channel a 
finite-state Markov channel. The a posteriori probability of 
a codeword x e X'^ given an output y e 3^^ is 



Pix\y) 



7 E n/(^5jn^(3''i^<,^/) 

" '=1 
N-l 

^o(«i) n^(*'+i \yi^^'^^') 



i=l 



N-l 



Similarly to the previous subsection, let E[ ] denote the ex- 
pectation with respect to e obeying 

Piy) ^ L L Un^da)Uw{yi I X,-,.,) 

N-l 

1=1 

As in the previous subsection, we consider (|5]l with Zq re- 
placed by 2^^. Then, one obtains 



E[Z"] = 



{k) (k) 



k=0 



1=1 \yeyk=o / 
Let T„ C S"^^ be the largest set such that for any S2 E %, 

LLuIl W{y\x24^)V{s['^ \y,-24^)] >0. 
s,eT„yeyk=o \x2ex ) 

In order to use Sanov's theorem [J5J from the method of 
types for Markov chain IIT6HT8I . we assume that the Markov 
chain on Tn defined by the transition probabilities 



is irreducible for each n E {0,1,2,...}. Then, we obtain 
the following maximization problem like LDPC codes for 
memoryless channel Q, 

lim ^\0gE[E[Z"]]=sup\HiXi,Si\X2,S2)-lH{XuSi) 

+ -//(f/i, . . . ,f/„ri, . . . ,r,) + - { iogn/([/«) ) 

\ A:=0 / 



log L n^(3' o^(5r b^^f ,5^) 

\ \y&yk=0 / 



R. 



(8) 



Here, Xi and X2 are random variables on X"^^. Si and ^2 
are random variables on % or equivalently on 5"^' . f/, and 
Ti are random variables on X"^^ and respectively, for 
i E {!,..., r}. The notation (•) denotes the expectation with 
respect to the random variables. The supremum is taken 
with respect to {Xi , 5i ,^2, ^2) and (t/i t/r, T, ) on 

the following conditions 

• {Xi , Si ) and (^2, '^2) have the same distribution 

• {Xi , Si ) and {Uk, Tk) have the same distribution 
where K denotes the uniform random variable on a set { 1 , ... , 
By using the variational method ||T9l , £j, we obtain the fol- 
lowing lemma. 

Leimna4. For n G {0,1,2,...}, 

lim -logE[Z"] = max 

N -^00 N (m^^f ,m^,^f ,mR^v .mL^s ) 



;=1 



logZw - logZv + - logZf - / logZe 

r 



R 



where TZ denotes the set of saddle points of the function for 
which the maximum is taken, and where 



Lemma 5. For n = 0, mR_>.v(ji:,s) is uniform. Hence, the 
stationary condition is given by (|2]l and 



2w:= L ij:nw{y\4'A>{s[''\y,4\4^) 

xi,si.X2.S2 \yeyk=o 

■ wr^v (jt 1 , Si )mf_^v (-tl )'otl^s («2)'«f-^v (-^2)' 

Zv ^mR^v(-«^,«)»JL^s(«)»Jf-^v(-»^)' 

x.s 

Ze : = ^ 'Mf-^v (-t ) OTv^f {x ) . 
a: 

77ie saddle point equations are 

m^^fix) oc ^mR^y{x,s)mi^^s{s)^ mf^v{x)'^^ 



mR^^{xi,si)mf^,,{xi)' 



Jfl.si \y€yk=0 



•mL^s(s2)wf_^v(-C2)'- 

//ere, f/ze domains of sums x,Xi,X2 G X"^^, s,Si,S2E S"^^ 
are omitted. 

Although a proof of this lemma is omitted for lack of 
space, the derivation is similar to the memory less case in Q . 



mL^sis)°^ 52 E l-^2'*2)^(« l3',-^2,i2) 
X2.S2 \ysy J 

•«^L^s(i2)wf-^v(-«2)'- 



(9) 



Proof. By considering (O, one can confirm that X\ and 
(also X2 and ^2) should be independent for n = 0. As a con- 
sequence, it turns out that mR^vlj^,*) is also an independent 
measure (This part relates to the proof of Lemma|4] Hence, 
the proof is omitted for lack of space). Then, Y^x^x '"R^v ('«^, s) 
should be uniform, which is the unique right eigenvector 
corresponding to the eigenvalue 1 of the stochastic matrix of 
irreducible Markov chain Qo- Then, one straightforwardly 
obtains that Y^^f^s '"R^v (-^ , s) is also uniform. □ 

For finite-state Markov channels, (|9]l implies that wl^s (*) 
is the stationary distribution of the Markov chain V (ii | st). 
The RS assumption for this problem is 



mR-^^{x,s) =mR^v(-«o,io) / WMR-^^{xi,Si) 
] •d1'(MR^v |xo,^o) 

OTL^s(«) =«iL^sC?o) / n^L^s(i,-)d^(ML^s I so)- 

1=1 

Similarly to the previous subsection, we obtain the RS solu- 
tion as follows, which is the main result of this paper. 
Theorem 6. The RS solution is given by dlOb . The saddle 
point equations are (|2]l, (|9|l and (I13l l. 



Wv^f(Ac) =OTv^f(jco) / ]~[Mv^f(jc,)d4>(Mv^f I xo) 

■' 1=1 

mf^y{x) ^mi^y{xQ) I Y\Mi^v{xi)d^{Mi^y \xq) 



extr.< 2- mL^sisi);^ ^ 2- W{y \ X2,S2)V {si \ y,X2,S2) 

d^(MR^v |^i,si)d'P(ML^s |*2)ncl4>(M<_^';| |xi)nd4>(M^!;'] 1^2) 

(=1 !=1 

•logf £ w(3'|x2,s2)v(si |y,^2,^2)MR^v(^i,^i) (n^^i;'](^i) Pl^^C^2) (n^|!;(!(^2) 



*y(A:i,il,-t2,S2)G'^X'SxA'x5 \'=1 / \'=' 

?Mf-^v(-x)' 

{x,s)eixxs) 

d^(MR^, I x,s)d*(ML^, I s)lld^M^l^ I x)log ( £ Mr^,{x,s)Mi^^,{s)IIm^^^{x) 

'■=1 \(x,i)GA'x5 1=1 

'^xGA''^ ■^-cGA"/ l-'^Jll,=l'"v^fl-*^ij ,=1 \xeX' i=l / 

; £ mf_,v(^)mv^f(x) ^^^^^^^ | x)d<I>(Mf_,v | x)log ( £ Mf_,,(x)Mv^f(;.)) 1 
xex^Lxexm^Yixjmy^fix} J \^ /I 



(10) 



Although the saddle point equations in f6l are obtained 
by simplifying Theorem|6]for finite-state Markov channels, 
they are omitted in this paper for lack of space. Similarly to 
the memoryless case, we have to deal with IRSB effect gen- 
erally. Unfortunately, the frozen solution does not generally 
exist. The necessary and sufficient condition of existence 
of the frozen solution is that for any y ^ y, when variables 
X2,S2 are fixed, at most one value si exists such that 



Wiy\x2,S2)V{si \ y,X2,S2)>0 



(11) 



and when si is fixed at most one pair of values {x2,S2) sat- 
isfying (fTTT l exists. This condition is called hard constraint 
in lfT4ll . When a state depends only on the previous state 
or input, we can obtain a simpler expression of RS solution 
than Theorem|6] and the necessary and sufficient condition 
for the existence of the frozen solution becomes weaker 

4 Numerical calculation 

While Theorem|6]gives the expected conditional entropy 
on general Markov channel, e.g., trapdoor channel, gener- 
ally we have to deal with densities of messages similarly to 
the memoryless case [Ti |, making implementation of the 
density evolution involved. In [20|, generalized erasure 
channel is defined as a channel where density evolution can 
be described by one parameter As the simplest general- 
ized erasure channel, the dicode erasure channel (DEC) is 
introduced. For A" = 5 = {0, 1} and 3^ = {-1,0, 1,*}, the 
DEC(e) is defined as 



Wiy\x,s) 



1 



3' = 

y- 



-x — s 
- * 



V{s' \y,x,s) = I, fors'=x 

for e e [0,1]. For the DEC, mL^,{s) = 1/2. We modify 
Theorem|6]for the case where the state only depends on the 
previous input, on the basis of which the RS solution for the 
DEC(e) is obtained without dealing with densities as 

lim -E[E[logZ]] = extr 

2(£loge + (l-£)log(l-e)) 

— Cr^v + (1 ^ £)£l^s + 2/ef_^v — seR^vEf-^v 



eLv ( e + ^(1 - e)eR^veLv j + " 



-eloge- (1 -e)log(l -e) + 



log2 

Er^v + (1 - e)eL^s + l£f^v 
log 2 



-(1 - (1 - ev^f)')log2 + - (1 - ef_^v)(l - ev^f))log2 



where the saddle point equations are 



l-(l-ev- 



Ev^f = e/_^J,eR^v ( e + :^(1 - e)eL- 



El 



£ + 2(1^ e)eR^vef-^v 



(12) 



mf^,{x)<f{Mf^,\x)^-y f TTTvfr ^WTT/ n d<P{Ml!^,\xi) 

^ f Lxex-Xxt fix) Ui^kMy^fjxi) 
\ Lxexrf{x)Ui^kM,^f{x^) 

4>(M,^f I x) = ^ mL^,(.) /d^(MR^v I x,s)d^{M^^, \ s)lldMMl^^ \ x) 

(I.e5MR^v(x,^)ML^s(^))nt!A^^l(x) ^ 



Lix,s)eXxsMR^. {x, ^)Ml^s (s) nt } {x) J 
^{M^^,\x,s) = Y. "''^'^'''^', W{y\x,s)V{s,\y,x,s) f d^(M^^J xi,.i)rid4>(M« J xi) 

I I(.v„.,)eA'x5<^v(^i-*i)(rt=iM<^,(xi))w(y|x,.)y(ii \y,x,s) 
■ S Mr^v, 

\ L^{x[,si.X2;S2)&XxSxXx 

sM'^_^^{xi,si)[Y\'i^iM'^^^{xi)]W{y\x2,S2)V{si \ y,X2,S2) 
ix2..s2)exxs Lx2ex'nf^v[X2) ^.^y J 



I 



1=1 



L(x2.s2)sXxsMi^,{s2) (u'i^iMl^^{x2))w{y\x2,S2)V{s\y,X2,S2y 

^{x2,S2)^XxS MUsi^2) {U^MiM)) Wiy\x2,s2) 

(13) 
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Figure 1 ; The expected conditional entropy of LDPC codes on 
the DEC calculated by the replica method. The results for (2,4), 
(3,6), (4,8), (5,10) and (6,12) regular ensembles are plotted. 

This is equivalent to the density evolution of the joint itera- 
tive decoding, which we call the belief propagation (BP) de- 
coding, on the DEC(e) EO). For the DEC(e), the expected 
conditional entropy is 

lim ^E[H{X I Y)] - f lim -J-E[E[logZ]] 

- eloge - (1 - e)log(l - e) ) 

J log2 

r- 1 

= -lEf^y + eel^s + 1 (1 - (1 - ef-^v)(l - ev^f)) 

r 

where fif-^v, £v^f and El^s satisfy (fT2l l. 

In Fig. [1] the results for (2,4), (3,6), (4,8), (5, 10) and 
(6, 12) regular ensembles are shown. When e is small, there 
exists only a trivial saddle point, which always gives ex- 
pected conditional entropy. A non-trivial saddle point ap- 
pears above BP threshold. However, similarly to the mem- 
oryless case, except for (2,4)-regular ensembles, the non- 
trivial saddle point yields negative expected conditional en- 
tropy for some regions as in Fig. [T] In the region, while 
BP decoding fails, MAP decoding succeeds since the trivial 
saddle point should be chosen for the non-negative condi- 
tional entropy. Above the MAP threshold, the non-trivial 
saddle point exhibits positive expected conditional entropy. 
As degrees increase, the BP threshold decreases while the 
MAP threshold increases. In contrast to the memoryless 
case, the BP threshold of (2, 4) -regular ensemble is higher 
than that of (3,6)-regular ensemble. The BP threshold and 
the MAP threshold for (3,6)-regularensemble is aboutO. 56891 
and 0.638 65, respectively. It implies that the upper bound 
of the MAP threshold in 121] is tight. Note that the Shannon 
threshold of rate-half codes is e = (1 + Vvf)/S « 0.64039. 
Similar figures for symmetric memoryless channels and asym- 
metric memoryless channels can be found in Il22l and 101, 
respectively. 

The method in this paper can be generalized to other 
models, e.g., IRA/ARA LDPC codes fTl\, CDMA on a 
channel with memory, compressed sensing of a Markov source, 
etc. 
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